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FOREWORD 


The  Presidio  of  Monterey  (POM)  Field  Unit  of  the  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  Training  Laboratory  performs 
research  and  development  in  the  areas  of  collective  ti  aining  and  unit 
performance  measurement.  Of  special  interest  to  the  POM  unit  is  research  in 
the  area  of  combat  performance  measurement  and  the  translation  of  those 
measurement  results  into  training  and  doctrine  implications  for  the  Army. 

The  National  Training  Center  (NTC)  at  Fort  Irwin,  California,  presents  an 
excellent  opportunity  to  provide  this  performance  data  on  Army  battalion  task 
forces  because  of  the  high-quality  combat  simulation  and  extensive  database 
available  for  analysis.  The  research  concept  presented  in  this  document 
provides  the  framework  for  the  development  and  application  of  an  analytical 
model,  or  structure,  which  directs  the  research  effort  toward  the  mission 
critical  events  to  be  observed  at  the  NTC,  the  development  of  their  standards 
of  performance,  measures  of  performance,  and  measures  of  effectiveness. 

The  operationalization  of  this  research  concept  is  designed  to  provide  a 
study  of  the  relationship  between  the  performance  of  certain  critical  mission 
tasks  and  the  mission  outcomes.  Products  emerging  from  this  research  effort 
will  be  in  the  form  of  NTC-specific  Army  Training  and  Evaluation  Programs 
(ARTEPs)  and  observer  guides  which  direct  performance  evaluators  to  the 
critical  events  which  directly  impact  upon  the  mission  outcomes.  Further, 
development  of  the  concepts  and  methodologies  which  are  operationalized 
through  this  research  effort  holds  great  promise  for  the  evolutionary  develop¬ 
ment  of  standardized  Army-wide  performance  measurement  systems.  The  Army's 
capability  for  determining  unit  readiness  could  thereby  be  improved  through 
the  measurement  of  high-payoff  training  events. 

This  research  effort  described  in  this  report  was  monitored  by  the  POM 
Field  Unit.  The  research  task  which  supports  this  mission  is  entitled  Field 
Feedback  from  National  Training  Center  to  Improve  Collective  and  Individual 
Training  and  is  organized  under  the  "Maintain  Force  Readiness"  program  area. 
This  research  effort  was  sponsored  by  the  Combined  Arms  Training  Activity 
(CATA)  under  the  Letter  of  Agreement  entitled  National  Training  Center  (NTC) 
and  Unit  Home-Station  Training  and  Feedback  System,  dated  16  September  1985. 
The  CATA  Lessons  Learned  Division  was  briefed  on  the  information  in  this 
document,  and  indicated  their  intention  to  make  use  of  the  results. 
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A  RESEARCH  CONCEPT  FOR  DEVELOPING  AND  APPLYING  METHODS  FOR  MEASUREMENT  AND 
INTERPRETATION  OF  UNIT  PERFORMANCE  AT  THE  NATIONAL  TRAINING  CENTER 

EXECUTIVE  SUMMARY 


Requirement: 

^The  purpose  of  this  report  is  to  present  a  research  concept  for  developing 
and  applying  methods  for  measuring  and  interpreting  unit  performance  at  the 
National  Training  Center  (NTC).  The  valid  measurement  of  unit  combat  effec¬ 
tiveness  during  peacetime  has  been  a  long-term  Army  goal.  The  NTC  provides  an 
excellent  opportunity  to  provide  this  performance  data  on  Army  battalion  task 
forces  because  of  the  high-quality  combat  simulation  and  extensive  database 
available  for  analysis. 


Procedure: 

A  conceptual  systems  model  for  viewing  task  force  performance  was 
developed.  In  order  to  operationalize  this  model,  two  measurement  systems  are 
to  be  simultaneously  developed.  One  system  will  be  the  measurement  of  mission 
results  and  the  developntent  of  mission  effectiveness  criterion  variables.  The 
second  measurement  system  will  be  the  measurement  of  the  performance  of 
critical  mission  tasks  and  the  effectiveness  of  that  performance. 


Findings: 

The  approach,  methodology,  and  procedures  for  developing  a  unit  perfor¬ 
mance  measurement  system  are  detailed.  The  milestones  and  personnel  require¬ 
ments  and  expected  research  products  are  also  presented.  The  system  which  is 
presented  takes  a  systemic  approach  to  task  force  performance  measurement  and 
provides  the  basis  for  data-gathering  activities  at  the  NTC  to  support  both 
training  and  off-line  analyses  of  "Lessons  Learned." 


Utilization  of  Findings: 

'j 

The  measurement  systems  described  in  this  research  concept  are  now  being 
developed.  The  strawrran  measurement  systems  will  be  tested  at  the  NTC, 
evaluated,  and  refined  for  operational  use.  The  concepts  and  methodology 
have  implications  for  use  by  the  Combined  Arms  Center  and  schools  in  develop¬ 
ment  of  Army  Training  and  Evaluation  Programs  (ARTEPs)  and  other  training 
support  materials. 
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A  RESEARCH  CONCEPT 
FOR  DEVELOPING  AND  APPLYING  METHODS 
FOR  MEASUREMENT  AND  INTERPRETATION  OF  UNIT  PERFORMANCE 
AT  THE  NATIONAL  TRAINING  CENTER 


OVERVIEW 

Purpose 

The  primary  purpose  of  this  report  is  to  present  a  research  concept 
for  developing  and  applying  methods  for  measuring  and  interpreting  unit 
performance  at  the  National  Training  Center  (NTC).  It  discusses  the 
background,  methods,  procedures,  milestones,  and  resource  requirements 
for  accomplishing  the  task. 


Introduction 


The  valid  measurement  of  unit  combat  effectiveness  during  peacetime 
has  been  a  long-term  Army  goal.  Of  particular  interest  is  the  transla¬ 
tion  of  unit  performance  results  into  training  and  doctrine  implications 
for  the  Army  (although  such  performance  results  may  also  provide  feedback 
on  the  equipment,  manning  and  organizational  structure(s)  of  the  type  of 
units  being  studied).  The  NTC  presents  an  excellent  opportunity  to  pro¬ 
vide  this  performance  data  on  Army  battalion  task  forces  because  of  the 
high-quality  combat  simulation  and  extensive  database  available  for 
analysis.  The  use  of  this  database,  however,  requires  an  analytic  frame¬ 
work,  or  model,  which  directs  the  research  effort  toward  the  mission- 
critical  events  to  be  observed,  how  they  are  to  be  recorded  and  analyzed, 
and  their  standards  of  performance,  measures  of  performance,  and  measures 
of  effectiveness. 


Research  Requirements 

With  the  above  understanding  and  appreciation  for  performance  analy¬ 
ses  at  the  NTC,  the  Army  Research  Institute  (ARI)  is  conducting  research, 
sponsored  by  the  Combined  Arms  Training  Activity  (CATA),  on  the  develop¬ 
ment  and  application  of  concepts  and  methods  for  measurement  and  inter¬ 
pretation  of  unit  performance  at  the  NTC.  This  effort  addresses  two  pri¬ 
mary  requirements.  One  is  to  develop  a  system  for  assessing  and  explain¬ 
ing  the  effectiveness  of  unit  performance  at  the  NTC.  Such  a  system, 
when  operationalized,  would  permit  determination  of  training  and  doctrine 
strengths  and  weaknesses.  It  also  would  allow  for  expanded  research  on 
predictor-criterion  studies  regarding  unit  performance.  The  second 
requirement  involves  the  development  of  two  by-products  of  this  measure¬ 
ment  system.  These  products  are:  (1)  an  ARTEP-like  document  for  the  NTC 
which  presents  the  Tasks,  Conditions,  and  Standards  in  clear,  unambig¬ 
uous,  and  descriptive  terms  such  that  performance  of  the  units  can  be 
uniformly  measured  using  the  performance  data  generated  at  the  NTC;  and 
(2)  observer  guides  on  selected  issues  which  would  assist  in  operational¬ 
izing  the  ARTEP  as  a  training  assessment  tool.  These  two  requirements, 
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then,  provide  the  focus  for  the  technical  approach  -  one  that  is  punctu¬ 
ated  with  requirements  for  military  expert  input,  non-armchair  field 
visits,  and  extensive  doctrinal  research  in  order  to  provide  realistic 
and  meaningful  measures  of  performance. 


METHOOS  AND  PROCEDURES 


General  Concept 

The  key  conceptual  requirements  in  this  research  effort  are  in  the 
terms  measurement  and  interpretation  of  unit  performance  at  the  NTC.  The 
ultimate  purpose  for  this  understanding  of  unit  performance  under  combat¬ 
like  conditions  during  peacetime  is  to  determine  its  training,  doctrine, 
equipment,  and  organizational  implications  thereby  allowing  the  Army  to 
better  prepare  itself  for  combat.  Our  technical  approach,  therefore,  is 
to  use  the  explanatory  paradigm  which  is  concerned  with  explaining  per¬ 
formance  results.  Thus,  explanatory  variables  must  be  identified  as  to 
their  role  in  relation  to  the  phenomenon  of  study  and  hence  require  some 
theoretical  or  conceptual  underpinnings.  Because  this  research  effort 
concerns  performances  of  organizations,  i.e.  battalion  task  forces,  we 
turned  to  organizational  theory  to  provide  us  with  a  focus  for  shaping  a 
conceptual  model.  Within  the  last  decade  there  seems  to  have  been  a 
shift  in  emphasis  away  from  management  and  structures  of  organizations 
towards  an  investigation  of  the  "systems"  of  organizations  —  the  study 
of  inputs,  processes,  outputs,  feedback,  the  environment  —  and  how  they 
relate  to  each  other.  This  conceptual  view  of  organizations  -  a  rela¬ 
tionship  of  subsystems  -  seemed  most  appropriate  for  this  explanatory 
research  effort  and,  therefore,  was  selected  as  the  conceptual  framework 
for  this  study.  A  general  systems  model  is  shown  at  Figure  1. 


Figure  1.  General  systems  model. 


In  relating  the  research  task  at  hand  -  i.e.  measuring  unit  perfor¬ 
mance  at  the  NTC  -  to  the  systems  model ,  we  are  able  to  transform  the 
systems  components  into  NTC  and  NTC-related  subsystems  as  shown  in  Fi¬ 
gure  2,  next  page. 
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Figure  2.  Systems  model  for  viewing  task  force 
performance. 


With  this  conceptual  model  in  hand,  we  address  its  operationaliza¬ 
tion  in  response  to  the  research  requirements. 

Operationalizing  the  Concept 

Our  general  approach  in  operationalizing  the  concept  will  be  to  si¬ 
multaneously  develop  two  measurement  systems  under  two  separate  tasks. 
One  system  will  be  the  measurement  of  the  mission  results  and  the  devel¬ 
opment  of  mission  effectiveness  criterion  variables.  The  second  measure¬ 
ment  system  will  be  the  measurement  of  the  performance  of  critical  mis¬ 
sion  tasks  and  the  effectiveness  of  that  performance.  Used  together  in  a 
relational  mode,  these  systems  are  intended  to  measure  and  explain  unit 
performance  effectiveness  which  is  the  first  research  requirement.  The 
fulfillment  of  this  first  requirement  will  then  allow  for  the  development 
of  the  NTC-specific  ARTEP  and  the  observer  guides  which  constitute  the 
second  research  requirement.  The  methodology  for  accomplishing  these 
requirements  are  discussed  next. 
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Requirement  1:  Development  of  a  System  for  Measuring  and  Explaining  Unit 
Performance  Effectiveness  at  the  NTC. 

Task  1:  Development  of  combat  mission  effectiveness  criterion  vari- 
ables.  This  task  addresses  the  "output"  subsystem,  or  NTC  Mission  Re¬ 
sults.  The  initial  analysis  effort,  which  will  become  the  cornerstone 
for  this  study,  will  be  directed  at  developing  the  effectiveness  criteria 
for  each  mission  conducted  at  the  NTC.  We  believe  that  in  order  to  pro- 
vide  any  meaningful  feedback  in  terms  of  training, .  doctrine,  etc.,  as 
reflected  by  the  performance  results,  the  overall  mission  effectiveness 
must  be  established.  This  notion  is  depicted  in  Figure  3. 


INPUT 


thnu-put 


OUTPUT 


Figure  3.  Developing  feedback  based  on  mission 
effectiveness  criteria. 


This  approach  is  also  consistent  with  the  need  to  develop  a  complete 
model  of  performance  within  the  explanatory  framework,  and  with  the  prior 
literature  such  as  Roberts  (1980)  who  stresses  that:  "one  cannot  move 
very  rapidly  to  the  independent  variable  side  of  the  unit  performance 
problem  until  questions  of  effectiveness  are  better  addressed."  Finally, 
but  no  less  important,  is  that  this  conceptualization  recognizes  the 
uniqueness  of  the  individual  missions  in  terms  of  their  requirements, 
difficulty,  and  accomplishments.  Thus,  measures  of  mission  effectiveness 
can  be  derived  for  each  battalion  task  force  on  a  mission-by-mission 
basis.  After  all,  the  accomplishment  of  the  tactical  mission  is  the 
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ultimate  purpose  of  a  combat  unit.  Therefore,  the  unit's  effectiveness 
should  be  measured  in  terms  of  its  mission  results,  a  notion  that  has 
strong  support  by  military  scholars  such  as  Sarkesian  (1982)  in  his 
edited  book  on  combat  effectiveness. 

The  methodology  for  fulfilling  this  first  task  is  through  the  under¬ 
taking  of  the  following  seven  subtasks: 

Subtask 

1.1:  Develop  Measurement  Model  for  Evaluating  Mission  Results  at 
the  NTC. 


1.2:  Establish  Mission  Performance  Conditions. 

1.3:  Set  Mission  Performance  Standards. 

1.4:  Determine  Measurement  Requirements  which  will  Result  in  Mea¬ 
sures  of  Mission  Performance. 

1.5:  Design  Mission  Criterion  Performance  Effectiveness  Indices. 

1.6:  Test  Measurement  System  Feasibility. 

1.7:  Prepare  Technical  Report. 

Implementation  procedures  for  each  subtask  are  discussed  next. 

Subtask  1.1:  Develop  measurement  model  for  evaluating  mission 

results  at  the  NTC.  The  ultimate  purpose  for  measuring  unit  effective- 
ness  is  to  determine  its  "causal"  antecedents  in  order  to  provide  direc¬ 
tion  for  training  and  doctrine  formulation  for  the  type  of  units  being 
assessed.  Implementation  of  this  kind  of  an  explanatory  research 
requires  the  development  of  a  model  or  conceptual  framework  which  can  be 
used  to  guide  measurement  development  and  statistical  testing.  In  this 
particular  subtask  the  requirement  is  to  develop  a  model  for  measuring 
mission  outcomes  and  assessing  performance  effectiveness,  and  will  be  a 
refinement  of  the  model  depicted  in  Figure  4,  next  page. 

As  can  be  seen  in  Figure  4,  the  structure  for  measuring  mission 
effectiveness  should  consist  essentially  of  four  parts:  Conditions,  Mis¬ 
sion  Standards,  Performance  Assessment,  and  the  resultant  Mission  Effec¬ 
tiveness  Criteria.  Each  of  these  four  parts  is  discussed  in  greater 
detail  below. 
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Figure  4.  Mission  effectiveness  measurement  system. 


Subtask  1.2:  Establish  mission  performance  conditions.  In  order  to 
establish  performance  standards,  we  must  first  establish  the  varying  con¬ 
ditions  under  which  the  task  forces  perform  their  respective  missions. 
To  accomplish  this  subtask,  the  battle  scenarios  used  for  each  of  the  NTC 
missions  (and  rules  for  preparing  variations)  will  be  collected, 
reviewed,  and  documented.  Our  experience  tells  us  that  no  two  scenarios 
are  exactly  alike  as  a  new  scenario  is  prepared  for  each  battalion  task 
force.  Therefore,  the  review  process  will  involve  60-90  scenarios.  How¬ 
ever,  we  anticipate  that  our  investigation  will  reveal  a  central  core  of 
common  conditions  that  would  have  such  a  significant  impact  upon  expected 
performance  results  that  they  must  be  taken  into  consideration  when 
establishing  performance  standards.  We  expect  that  these  conditions  will 
be  expressed  in  METT-T  (Mission,  Enemy,  Troops,  Terrain  and  Weather,  and 
Time)  terms  as  they  represent  the  major  factors  that  are  considered  by 
the  military  in  determining  courses  of  action  to  achieve  desired  results. 
These  critical  conditions  will  be  analyzed  and  specified.  The  Combined 
Arms  Training  Activity  (CATA)  will  be  asked  to  review  the  results  of  this 
subtask  for  suggestions  and  recommendations. 
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Subtask  1.3:  Set  mission  performance  standards.  In  order  to  deter¬ 
mine  the  effectiveness  of  a  unit,  its  performance  must  be  compared  to 
some  standard  or  desired  level  of  performance.  Such  standards  must  be 
acquired  from  subject  matter  expert  (SME)  judgments  for  each  of  the  task 
force  missions  and  related  missions  of  subordinate  elements  through  pla¬ 
toon  level,  as  a  function  of  the  scenarios  and  conditions.  These  stan¬ 
dards  must  be  expressed  in  measurable  dimensions.  They  will  be  reviewed, 
revised,  and  approved  by  CATA.  Figure  5  is  an  example  of  this  part  of 
the  measurement  system.  A  discussion  of  its  components  follows. 

(1)  Attribute:  Systems  theory  tells  us  that  we  should  analyse  the 
quality  of  organizational  output  in  terms  of  the  organization's  purpose. 
When  we  look  at  the  doctrinal  purposes  of  the  offense  or  defense,  we  find 
that  each  purpose  is  expressed  in  terms  of  Mission,  Enemy,  Troops,  Ter¬ 
rain  and  Weather,  and  Time  (METT-T)  factors.  Thus  we  find  the  considera¬ 
tion  of  METT-T  factors  to  be  not  only  consistent  with  military  thinking 
but  with  our  theoretical  base  as  well.  Also,  in  the  practical  sense,  we 
know  that  performance  data  from  the  NTC  are  available  for  analyzing  these 
attributes. 

(2)  Purposes  of  Mission:  Using  doctrine  from  the  appropriate  mili¬ 
tary  manuals  and  judgments  of  military  experts,  the  primary  and  secondary 
purposes  foreach  type  of  mission  conducted  at  the  NTC  will  be  identi¬ 
fied.  Former  NTC  cadre  members  and  CATA  personnel,  for  example,  could 
provide  the  military  expertise  while  FM  71-2J  would  be  the  oasic  doctri¬ 
nal  text.  The  Brigade  Operations  Orders  (OPORD)  which  are  on  file  in  the 
ARI  NTC  library  will  also  provide  valuable  input  to  this  process.  After 
determining  the  specific  mission  purposes  we  then  will  establish  the 
standards  for  achievement  of  these  purposes. 

(3)  Standards:  This  is  the  preeminent  and  most  difficult  task  in 
developing  the  mission  effectiveness  measurement  system.  We  obviously 
need  to  establish  mission  standards,  for  without  them  there  will  be  no 
way  of  determining  the  relative  merit  of  mission  results  which  in  turn 
would  prevent  an  analysis  of  the  strengths  and  weaknesses  of  the  antece¬ 
dents  of  those  mission  results.  While  the  outcome  of  military  missions 
are  situational ly  dependent  we  are  limiting  ourselves  to  the  NTC  where 
the  missions,  scenarios,  and  Division  Operations  Plans  are  standardized 
and  the  terrain,  OPFOR,  and  externally  imposed  stress  are  a  relative  con¬ 
stant.  The  standards  must  also  take  into  account  the  various  task  force 
configurations  and  organizational  equipment.  These  variations  include 
mechanized  versus  armor  task  forces  and  units  equipped  with  M1/M2  armor 
weapon  systems  versus  M60/M113  armor  systems  or  a  combination  of  these. 
We  therefore  expect  that  mission  performance  standards  can  be  established 
considering  NTC-specific  situations  and  consequently,  the  task  of  estab¬ 
lishing  mission  standards  becomes  a  manageable  endeavor. 
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Figure  5.  Mission  standards 


Initially,  we  intend  to  approach  this  task  through  a  review  of  the 
applicable  military  manuals  but  with  heavier  emphasis  on  expert  military 
judgments.  Former  Brigade  Commanders  of  the  NTC  rotation  brigades  will 
be  excellent  sources  for  this  information.  Procedurally,  we  probably 
will  need  to  visit  with  several  of  these  military  experts  to  develop  the 
scope  of  the  issues  surrounding  the  establishment  of  mission  standards, 
i.e.  institutional  biases,  understanding  of  the  task,  ways  to  best 
approach  military  experts  to  gather  the  data,  etc.;  in  other  words  we 
need  to  determine  how  to  best  sell  the  idea  of  establishing  mission  stan¬ 
dards.  Once  a  saleable  package  is  developed,  we  can  then  seek  consensus 
from  military  experts  on  mission  standards  through  a  process  such  as  the 
Delphi  technique.  Historical  NTC  data  will  also  be  used  to  empirically 
establish  a  distribution  of  performance  results  from  which  actual  mission 
results  could  be  compared  to  the  desired  standards  set  by  the  military 
experts.  Major  deviations  between  the  two  standard  setting  sources  will 
be  addressed  and  resolved,  probably  through  a  repeated  Delphi  process. 

Subtask  1.4:  Determine  measurement  requirements  which  will  result 
in  measures  of  mission  performance.  Once  the  dimensions  of  the  standards 
are  known  we  will  then  determine  the  essential  elements  of  information 
required  to  produce  MOPs  that  possess  the  same  metric  properties  of  the 
standard  so  that  direct  comparisons  can  be  made  between  them.  We  will  be 
tempered  in  our  approach  by  the  capability  of  the  assessment  system  to 
provide  the  required  subjective  and  objective  data.  It  must  be  a  fea¬ 
sible  and  practical  approach  and  the  results  closely  coordinated  with 
CATA.  Figure  6  displays  an  example  of  this  requirement.  A  discussion  of 
the  two  components  of  this  part  of  the  measurement  system  is  presented 
below. 

(1)  Data  Requirements:  Once  the  mission  standards  are  established, 
the  data  requirements  that  would  reliably  and  validly  represent  the 
dimensions  of  those  standards  will  have  to  be  established.  The  capabil¬ 
ities  of  the  current  in-place  data  sources  will  be  considered  as  a  prac¬ 
tical  matter  when  establishing  the  data  requirements.  These  sources 
include  digital  tapes,  OC  judgments,  MILES  engagements  (uninstrumented), 
Take  Home  Packages,  Audio  Tapes,  Operation  Plans,  Operation  Orders,  and 
After  Action  Reviews.  However,  new  data  sources  may  also  be  required. 
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(2)  Measures  of  Performance:  Now  that  the  candidate  MOPs  have  been 
identified  (in  (1)  above),  the  actual  unit  performance  can  be  measured 
and  recorded  here. 

Subtask  1.5:  Design  mission  criterion  performance  effectiveness 
indices.  At  this  point  the  effectiveness  of  the  performance  of  each  unit 
by  mission  can  be  assessed  by  using  the  measurement  structure  provided 
above,  i.e.,  comparing  the  MOP  with  the  desired  standard  score.  This 
information  would  probably  be  sufficient  for  the  military  units'  perfor¬ 
mance  assessment  efforts.  However,  for  research  purposes  these  data 
would  be  very  unwieldly.  Therefore,  since  these  assessments  will  provide 
multiple  scores  based  on  the  six  or  seven  basic  NTC  missions  and  as  many 
as  six  different  standards  per  mission  (see  Attributes,  Figure  4),  a 
method  will  be  developed  to  reduce  these  scores  into  a  more  manageable 
effectiveness  index  which  would  greatly  facilitate  the  research  efforts 
in  producing  correlational  studies  relating  predictors  to  outcomes. 
Figure  7,  next  page,  shows  an  example  of  this  part  of  the  measurement 
system.  We  cannot  overly  emphasize  the  use  of  this  part  of  the  system  as 
a  research  tool  only.  Each  component  of  the  above  measurement  structure 
is  discussed  below. 

(1)  Scoring  Protocol:  This  component  of  the  measurement  system 
will  provide  the  rules  for  converting  the  results  of  the  raw  scores  of 
the  various  effectiveness  attributes  into  an  effectiveness  index.  One 
way  to  do  this  would  be  to  compare  the  MOP  with  the  desired  standard 
score.  The  comparative  results  could  be  obtained  by  placing  the  perfor¬ 
mance  measurements  on  a  five-point  scale  in  which  the  desired  standard  is 
at  the  mid-point.  Thus,  the  raw  score  would  then  be  converted  to  a 
single  digit  which  reflects  the  relationship  of  the  performance  to  the 
desired  standard.  In  this  manner,  the  raw  scores  of  the  various  effec¬ 
tiveness  attributes  (terrain,  time,  resources)  which  have  differing 
metric  properties  can  be  standardized;  also,  the  single  digit  score  would 
be  a  value-based  score  (score  based  on  the  desired  level  of  performance) 
thus  allowing  for  relationships  to  be  made  between  effectiveness  as 
applied  to  a  standard  and  its  causal  antecedents.  After  weighting  the 
scores  based  on  the  relative  importance  of  the  effectiveness  attribute  to 
the  overall  mission  results,  the  scores  could  be  totalled  to  provide  a 
single  effectiveness  index. 

Another  method  for  producing  an  effectiveness  index  would  be  to  com¬ 
pare  the  MOPs  for  all  the  units  thus  establishing  a  distribution  of  per¬ 
formances  which  then  allows  for  a  comparison  of  the  effectiveness  of 
units  relative  to  each  other.  This,  however,  would  not  provide  an  index 
reflective  of  the  units  performance  effectiveness  based  on  the  desired 
level  of  performance.  This  method  would  entail  the  conversion  of  raw 
scores  (MOPs)  to  Z-scores  for  each  attribute;  the  scores  are  then 
weighted  to  provide  initial  equalization  of  the  contribution  of  each 
attribute  to  the  total  score  for  each  mission. 


12 


13 


(2)  Points  Scored:  This  will  simply  be  the  results  of  applying  the 
raw  data  to  the  scoring  protocol  resulting  in  a  single  digit  standardized 
score. 

(3)  Weight  of  Attribute:  We  postulate  that  the  contribution  of  the 
effectiveness  attributes  (terrain,  resources,  time)  to  mission  results 
vary  across  missions.  Thus,  the  relative  weights  of  each  attribute  to 
each  mission  must  be  determined  and  applied  to  the  points  scored,  i.e. 

(2)  above,  in  order  to  provide  the  relative  contribution  of  the  attri¬ 
butes  to  the  total  score  of  each  mission.  The  initial  weights  will  be 
derived  from  doctrinal  manuals  and  by  military  experts. 

(4)  Weighted  Score:  This  will  be  the  result  of  multiplying  (2)  and 

(3)  above.  The  total  of  the  weighted  scores  then  becomes  the  effective¬ 
ness  index. 

Sub  task  1.6:  Test  measurement  system  feasibility.  CATA  will  be 
requested  to  collect  observational  data  in  order  to  test  and  validate  the 
measurement  system's  feasibility.  The  results  will  be  used  to  refine  the 
system. 

Subtask  1.7:  Prepare  technical  report.  The  measurement  model  and 
methodology  developed  for  its  use  at  the  NTC  will  be  documented  in  a 
technical  report. 


Task  2:  Development  of  critical  task  performance  measurement  sys- 
tem.  In  Task  1  of  this  research  effort,  the  development  of  the  mission 
effectiveness  measurement  system  is  being  undertaken  to  provide  a  basis 
for  determining  the  contribution  of  the  NTC  organizational  processes  to 
the  mission  results.  To  perform  this  relational  study,  one  must  be  able 
to  measure  the  performance  of  these  organizational  processes.  That  is, 
we  postulate  that  in  order  to  explain  the  results  of  the  missions  per¬ 
formed  at  the  NTC,  one  must  investigate  the  task  force  processes  at  the 
NTC  as  predictors  of  mission  performance.  This  .notion  is  shown  in 
Figure  8. 


input 
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Figure  8.  Measuring  organizational  processes  at  NTC. 


In  a  sense,  these  factors  are  intermediate  performance  criteria 
(Roberts,  1980)  that  provide,  we  believe,  the  linkage  between  mission 
results,  and  home  station  factors.  Organizational  systems  theory  tells 
us  that  these  processes  would  consist  of  unit  and  individual  task  perfor¬ 
mances  as  well  as  other  factors  in  the  broad  areas  of  behavior  and  unit 
values.  (Figure  9,  next  page) 

In  this  research  project  we  will  confine  our  efforts  to  the  critical 
individual  and  unit  tasks.  In  particular,  the  individual  tasks  will 
include  critical  leadership  behaviors  which  impact  upon  unit  performance 
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Figure  9.  Organizational  processes  influencing  mission  results. 


on  the  battlefield.  (The  investigation  of  the  other  behaviors  and  unit 
values  and  their  relationship  to  mission  results  would  be  a  logical 
follow-on  to  this  study,  however).  These  critical  tasks  might  be  called 
the  "high-payoff"  tasks  which  should  be  emphasized  during  periods  of  unit 
training  to  insure  that  maximum  time  is  given  to  mastering  those  critical 
task  skills.  By  developing  an  understanding  of  the  relationship  of  task 
performance  to  the  effectiveness  of  the  overall  mission,  the  Army  can 
determine  the  critical  tasks  for  training  and  units  can  plan  training 
accordingly.  Therefore,  identifying  and  measuring  these  critical  tasks 
will  be  a  major  effort  in  this  study.  (As  an  added  utility,  the  identi¬ 
fication  of  these  tasks  can  have  immediate  and  practical  application  to 
the  NTC  in  the  form  of  a  specifically  developed  NTC-like  ARTEP  and 
observer  guides  which  direct  the  observers  to  the  ARTEP  tasks  with  given 
conditions  and  standards.  The  development  of  these  two  products  are  dis¬ 
cussed  later  in  Requirement  2  of  this  research  concept). 
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The  methodology  for  fulfilling  this  second  task  is  through  the 
undertaking  of  the  following  seven  subtasks: 

Subtask 

2.1:  Oevelop  a  Measurement  Model  for  Evaluating  Critical  Task 

Performance  at  the  NTC. 

2.2:  Analyze  and  Identify  Mission  Critical  Tasks  for  all  NTC 

Missions  from  Battalion  down  to  Platoon  Level. 


2.3 


2.4 


2.5: 

the  Measures 


2.6: 


2.7: 


Establish  Critical  Task  Conditions. 

Establish  Critical  Task  Performance  Standards. 

Determine  Measurement  Requirements  which  will  Result 
of  Critical  Task  Performance. 

Develop  Critical  Task  Effectiveness  Criterion  Measures. 

Prepare  Technical  Report. 


in 


The  implementation  of  these  subtasks  are  discussed  next. 

Subtask  2.1:  Develop  a  measurement  model  for  evaluating  critical 
task  performance  at  the  NTC.  In  order  to  measure  critical  task  perfor¬ 
mance  we  will  first  have  to  develop  a  structure  or  format  for  gathering 
and  analyzing  the  data.  It  will  be  used  to  guide  measurement  development 
and  statistical  testing  and  will  be  the  basis  for  the  establishment  of  a 
relational  database.  This  model  will  be  a  refinement  of  the  structure 
shown  in  Figure  10,  next  page.  It  will  consist  essentially  of  three 
parts:  Task  Standards,  Performance  Assessment  and  the  resultant  Task 
Effectiveness  Criteria  measure.  A  discussion  of  each  part  follows  under 
subtasks  2.2  through  2.6  below. 
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Figure  10.  Critical  task  measurement  system. 


Subtask  2.2:  Analyze  and  identify  mission  critical  tasks  for  all 
NTC  missions  from  battalion  down  to  platoon  level.  We  will  first  estab¬ 
lish  the  constructs  of  the  critical  tasks  by  unit  level.  This  effort 
will  be  the  framework  for  identifying  the  critical  tasks.  For  example, 
at  the  battalion  task  force  level  the  construct  may  be  the  seven  combat 
imperatives  described  in  FM  71-2J  which  would  then  lead  the  researcher  to 
the  critical  tasks  which  correspond  to  those  imperatives  and  which  affect 
the  battalion  operation  as  a  whole.  We  will  also  establish  delimitations 
for  tasks  by  unit  level  of  analysis  to  give  uniqueness  to  the  task 
levels,  so  as  to  avoid  confusion  or  overlap  between  them.  Thus,  battal¬ 
ion  level  tasks  may  be  those  activities  that  generally  state  what  the 
battalion  should  do  to  fulfill  the  combat  imperatives  without  reference 
to  the  specific  element. 

We  next  will  establish  what  these  tasks  are.  As  a  start  point,  we 
will  use  appropriate  doctrinal  and  training  publications  and  military 
experts  to  develop  a  list  of  tasks.  This  will  be  approached  from  a  top- 
down  rationale.  In  other  words,  we  will  first  establish  the  essential 


18 


RW 


mmmmmmmmmmmmmmmrmmmm  n— — ww i  wmmm mmm mmmm 


wrr 


tasks  for  a  battalion  task  force  undergoing  the  six  to  seven  types  of 
missions  at  NTC,  then  dendritical ly  identify  the  tasks  that  the  task 
force  elements  down  to  platoon  level  must  perform  in  order  to  allow  those 
battalion  mission  critical  tasks  to  be  accomplished.  These  critical  unit 
tasks  will  also  provide  the  context  for  determining  critical  leadership 
tasks.  That  is,  each  critical  unit  task  will  be  examined  as  to  the 
leadership  tasks  that  will  be  required  in  order  for  that  unit  task  to  be 
accompl ished. 

Subtask  2.3:  Establish  critical  task  conditions.  As  discussed  ear¬ 
lier  Ijnder^ubUsk  TiT15i_rrTTTi-TiTabTTshTrig~The"  conditions  which  we 
believe  will  present  such  uniquely  challenging  situations  to  the  battal¬ 
ion  task  forces  that  they  must  be  considered  when  establishing  the  mis¬ 
sion  performance  standards.  These  conditions  will  probably  be  expressed 
in  terms  of  METT-T  factors.  Naturally,  these  mission-outcome  oriented 
conditions  must  be  considered  when  establishing  desired  critical  task 
outcomes  as  they  no  doubt  will  influence  the  performance  of  those  tasks. 
However,  there  will  also  be  task  conditions  that  must  be  considered. 
These  conditions  are  unique  to  the  task  and  may  be  viewed  as  representing 
the  situation  under  which  each  task  is  to  be  performed.  For  example,  in 
the  "Defend  in  Sector"  mission  a  critical  battalion  level  unit  task  may 
be  to  "initiate  direct  fire  engagements  at  maximum  effective  range  with 
HAWs  and  tanks."  The  situational  condition  under  which  this  task  would 
be  accomplished  could  be,  "The  battalion  TF  has  occupied  initial  posi¬ 
tions  forward  in  the  sector.  OPFOft  attacks  in  the  battalion  TF  sector." 
The  NTC  scenarios  and  NTC  SMEs  will  again  serve  as  a  useful  informational 
sources  for  these  conditions. 

Subtask  2.4;  Establish  critical  task  performance  standards.  The 
standards  of  performance  of  the  critical  tasks  must  next  be  developed. 
We  do  not  expect  to  have  as  much  difficulty  initially  with  this  as  with 
developing  the  standards  for  the  mission  performance  measurement  system 
(Task  1)  because  the  doctrinal  and  training  manuals  are  fairly  clear  as 
to  standards  of  performance  for  critical  tasks.  For  example,  the  task  of 
engaging  enemy  armor  at  a  maximum  range  gives  the  researcher  a  clear 
indication  of  the  standard.  Employing  scouts  forward  of  the  FEBA  is 
another  example  of  a  clearly  worded  standard  (although  the  distance  for¬ 
ward  of  the  FEBA  would  need  to  be  established).  The  conditions  used  for 
establishing  the  standards  will  be  based  on  those  conditions  identified 
in  Subtasks  1.2  and  2.3  of  this  research  effort.  As  a  start  point,  doc¬ 
trinal  manuals  such  as  FMs  7-7J,  71-1J  and  71-2J,  will  be  used  to  estab¬ 
lish  these  standards.  However,  we  must  go  beyond  simply  "armchairing" 
the  establishment  of  these  standards.  In  addition  to  reviewing  the  doc¬ 
trinal  material,  we  must  consult  with  the  military  experts  who  have  been, 
or  are  currently,  on  the  ground  at  the  NTC,  observing  the  training  of  the 
battalion  task  forces.  Their  insights  gained  from  observing  numerous 
unit  performances  should  prove  to  be  invaluable  to  this  standard  setting 
effort.  Other  SME  judgments  will  be  obtained  through  visits  to  Forts 
Benning  and  Knox.  Also,  in-house  military  expertise  will  be  used  as  doc¬ 
trinal  resources.  During  the  process  of  establishing  these  standards  we 
will  consider  the  capabilities  of  the  present  and  possible  future  NTC 
data  collection  system  to  provide  us  with  the  required  data  elements  that 
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reflect  performance  in  terms  of  the  standards'  parameters.  In  other 
words,  the  standards  must  consist  of  data  elements  that  can  be  feasibly 
collected.  Therefore,  to  assist  us  in  this  endeavor,  we  will  establish 
the  data  requirements  and  candidate  measures  of  performance  for  e-ch  cri¬ 
tical  task.  These  measurement  properties  will  also  provide  the  basis  for 
the  development  of  the  observer  guides  to  be  developed  under  Require¬ 
ment  2  of  this  research  concept.  This  portion  of  the  measurement  system 
is  discussed  below. 

Subtask  2.5:  Determine  measurement  requirements  which  will  result 
in  the  measures  of  critical  task  performance.  There  are  two  basic 
requirements  here.  One  is  to  develop  the  data  requirements  which  will 
provide  performance  measures  in  terms  of  the  dimensions  of  the  standards. 
The  second  requirement  is  to  provide  the  measurement  system  with  a  capa¬ 
bility  for  recording  the  actual  measure  of  performance. 

Subtask  2.6:  Develop  critical  task  effectiveness  criterion  mea¬ 
sures.  This  portion  of  the  critical  task  measurement  system  is  being 
undertaken  in  order  to  simplify  the  research  efforts  in  relating  the  per¬ 
formance  of  the  critical  tasks  to  the  mission  performance  criteria 
indices.  This  analytical  tool  will  allow  the  researcher  to  reduce  the 
multitude  of  performance  measurements  into  standardized  effectiveness 
scores  which  can  more  readily  be  correlated  to  the  mission  results  mea¬ 
sured  under  Task  1  of  this  research  requirement.  Each  component  of  this 
section  of  the  measurement  system  is  discussed  next. 

(1)  Scoring  Protocol:  We  could  simplify  and  standardize  the  mea¬ 
sures  by  computing  Z-scores  for  each  of  the  measures  or  by  establishing  a 
five-point  scale  with  the  standard  score  being  at  the  mid-point.  This 
approach  is  similar  to  that  taken  in  establishment  of  the  mission  effec¬ 
tiveness  measurement  system.  The  5  point  scale  scoring  protocol  is  more 
attractive  than  the  Z-score  method,  however,  because  of  the  value-based 
meaning  of  the  single  digit  score  (value  being  a  score  with  respect  to 
the  desired  standard). 

(2)  Points  Scored:  This  would  simply  be  the  results  of  applying 
the  raw  data  to  the  scoring  system  (protocol)  resulting  in  a  single  digit 
standardized  score. 

Subtask  2.7:  Prepare  technical  report.  A  technical  report  will  be 
prepared  which  documents  the  approach  and  methodology  used  in  developing 
the  critical  task  measurement  system  and  the  final  results  of  that 
effort. 
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Requirement  2:  Development  of  NTC-Specific  ARTEP  and  Observer  Guides. 

As  mentioned  earlier,  the  development  of  the  above  performance  mea¬ 
surement  systems  will  allow  for  the  construction  of  two  by-products  of 
those  systems  -  an  NTC-specific  ARTEP  and  observer  guides  which  will 
serve  to  operationalize  the  ARTEP  as  a  training  assessment  tool.  Funda¬ 
mentally,  these  two  products  would  be  extracted  from  the  portions  of  the 
critical  task  measurement  system  shown  below  in  Figure  11. 


Figure  11.  NTC-ARTEP  and  observer  guides  extracted  from 
measurement  system. 


Task  1;  Develop  NTC-specific  ARTEP.  The  most  recent  ARTEPs  for 
battalion  task  force,  company,  and  platoon  level  training  will  be 
reviewed  for  accuracy  and  completeness  based  on  the  tasks,  conditions, 
and  standards  identified/developed  above  for  the  type  missions  conducted 
at  the  NTC.  Specific  gaps  or  potential  inaccuracies  will  be  presented  to 
CATA  for  review  and  comment.  We  will  also  present  a  strawman  format  for 
an  NTC-related  ARTEP.  Based  on  the  CATA  review  comments,  a  draft  Supple¬ 
ment  ARTEP  for  the  NTC  will  be  developed  incorporating  information  from 
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ft  LS  AnA  Vl  Jasks,  conditions,  standards  established  above. 
It  is  expected  that  the  observational  and  objective  data  on  unit  perfor¬ 
mance  obtainable  at  the  NTC  will  permit  the  use  of  a  less  "procedural- 
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TRAINING  AND  EVALUATION  OUTLINE 


WIT!  Battalion  T«lc  Force 


MISSION:  o.ftad  (J.H-3, 

task 

CONDITIONS 

3-YI-3-8 

The  battalion  TF 

Defend  In 

occupies  Initial 

Sector. 

positions 

forward  In  the 
sector.  OPFOK 
attacks  In  the 
battalion  TF 
sector. 


STANDARDS 


The  battalion  task  force: 

1.  Scout  platoon  detects  and 
reports  attacking  OPFOR  elements. 
The  scout  platoon  aalntalna 
contact;  assists  passage  of  the 
covering  force.  If  necessary;  and, 
on  order,  disengages  and  novas  to 
conduct  another  alsslon. 

2.  Engages  attacking  OPFOK  with 
supporting  fires. 

3.  Initiates  direct  fire 
eogageaents  st  eaxleue  effective 
range  with  HAVs  and  tanks. 

A.  Maneuvers  to  the  flanks  of 
OPFOK  avenues  of  approsch  to 
obtain  flanking  shots. 


URRINCn 
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chap  5. 
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Figure  12.  Sample  of  current  ARTEP. 
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Task  2:  Prepare  observer  guides.  A  diagnostic  tool  for  measuring 
the  performance  of  the  critical  tasks  outlined  in  the  ARTEP  must  be 
developed  so  as  to  operationalize  the  ARTEP  structure  developed  above. 
This  diagnostic  tool  will  present  the  required  assessment  data,  based  on 
the  ARTEP  standards  that  the  NTC  Observer  Controllers  (OCs)  can  collect 
when  assessing  unit/individual  performances.  In  essence,  it  will  serve 
as  an  observer  guide  in  assisting  the  OCs  in  performing  their  duties.  It 
will  help  to  insure  that  the  observers  are  uniformly  and  consistently 
assessing  the  high-payoff  activities  that  have  the  greatest  impact  upon 
unit  performance.  Their  observations  will  also  serve  to  validate  the 
NTC-related  ARTEP  as  a  meaningful  and  valuable  training  tool. 

The  basis  for  the  content  of  these  observer  guides  will  be  the  PER¬ 
FORMANCE  ASSESSMENT  portion  of  the  critical  task  measurement  system.  A 
format  for  displaying  and  recording  the  required  evaluation  data  will  be 
developed  with  the  underlying  principle  that  it  enhances  and  not  impedes 
the  duties  of  the  observers  at  the  NTC.  We  will  seek  the  advice  of  the 
NTC  Operations  Group  in  the  development  of  these  guides.  CATA  will 
review  and  approve  these  guides.  Finally,  the  feasibility  and  utility  of 
the  observer  guides  will  be  tested  by  CATA  through  application  at  the  NTC 
for  selected  issues.  A  report  of  the  results  of  the  field  test  of  these 
guides  will  be  prepared. 

Determining  Reliability  and  Validity  of  Measurement  System. 

At  the  end  of  the  developmental  sequence  described  above,  a  measure¬ 
ment  system  for  battalion  NTC  mission  performance  will  have  been  opera¬ 
tionalized.  The  system  will  have  been  tested  for  its  feasibility  under 
Subtask  1.6,  Requirement  1,  and  through  the  creation  and  utilization  of 
observer  guides  under  Task  2,  Requirement  2.  It  should  be  noted,  how¬ 
ever,  that  issues  of  reliability  and  validity  will  still  remain  to  be 
addressed. 

With  respect  to  reliability,  it  will  be  necessary  to  demonstrate 
that  the  observer  guides  and  other  data  gathering  instruments  can  be  used 
in  a  consistent  manner  to  gather  the  required  data.  To  accomplish  this, 
it  will  be  necessary  to  conduct  a  field  study  at  the  NTC  to  collect  suf¬ 
ficient  measurements  to  allow  for  the  empirical  investigation  and  calcu¬ 
lation  of  reliability  coefficients.  This  will  require  designing  a  data 
collection  plan  that  specifies  the  conditions  and  situations  under  which 
observations  will  be  taken  so  that  reliability  estimates  may  be  calcu¬ 
lated  appropriately. 

The  establishment  of  validity  of  the  system  involves  the  determina¬ 
tion  of  two  types  of  validity.  The  first,  content  validity,  will  largely 
be  addressed  and  satisfied  by  a  thorough  documentation  of  the  develop¬ 
mental  procedure  associated  with  the  measurement  system.  The  extensive 
use  of  Subject  Matter  Experts  and  doctrinal  references  will  lend  consi¬ 
derable  credibility  to  the  claim  of  content  validity  for  the  measurement 
system.  The  second,  construct  validity,  would  appear  to  be  more  relevant 
to  the  effectiveness  criterion  measures  portion  of  the  measurement  sys¬ 
tem.  The  establishment  of  this  type  of  validity  will  require  empirical 
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investigation  with  independent  estimates  of  mission  effectiveness.  As 
with  the  issue  of  reliability,  a  separate  data  collection  will  be 
required  to  allow  for  the  empirical  evidence  of  construct  validity. 

Thus,  additional  work  beyond  the  scope  of  this  research  effort  will 
be  required  to  establish  the  reliability  and  construct  validity  of  the 
measurement  system.  It  is  essential  that  this  work  be  performed  before 
the  new  measurement  system  is  routinely  used  at  the  NTC. 

SUMMARY 

The  purpose  of  this  research  is  to  develop  concepts  and  methods  for 
measuring  and  interpreting  unit  performance  at  the  National  Training  Cen¬ 
ter.  This  research  concept  presents  the  approach,  methodology  and  proce¬ 
dures  for  accomplishing  this  task.  The  milestones/personnel  requirements 
and  expected  research  products  are  also  presented  at  APPENDIXES  A  and  B 
respectively.  We  believe  this  concept  will  produce  a  measurement  system 
that  not  only  meets  the  research  objectives  but  one  which  will  provide 
distinct  advantages  over  the  present  system  inplace  at  the  NTC.  Figure 
13,  next  page,  capsulizes  these  advantages. 


24 


UNIQUE  CHARACTERISTICS 

TAKES  A  SYSTEMIC  APPROACH  TO 
PERFORMANCE  MEASUREMENT 

IS  OUTCOME  ORIENTED 

ESTABLISHES  MISSION  effectiveness 
CRITERIA 


IDENTIFIES  AND  MEASURES  PERFOR¬ 
MANCE  OF  TASKS  CRITICAL  TO 
MISSION  OUTCOMES 


PROVIDES  THE  BASIS  FOR  THE  CONTENT 
ANO  PROCESS  OF  PERFORMANCE  DATA- 
GATHERING  ACTIVITIES  AT  THE  NTC 
IN  SEARCH  OF  “LESSONS  LEARNED" 
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ADVANTAGE  OVER  PRESENT  SYSTEM 


ALLOWS  FOR  A  MORE  INTEGRATED  AND  COMPREHENSIVE 
CONSIDERATION  OF  VARIABLES  WHICH  SYSTEMATICALLY 
EFFECT  UNIT  PERFORMANCE. 

FOCUSES  MEASUREMENT  ACTIVITIES  ON  THE  PURPOSE 
FOR  UNIT  PERFORMANCES  AT  THE  NTC  •  I.E..  THE 
ACCOMPUSHMENT  OF  SPECIFIC  TACTICAL  MISSIONS. 


ESTABLISHES  STANDARDS  FOR  MISSION  OUTCOMES. 


ALLOWS  FOR  THE  IDENTIFICATION  OF  THE  STRENGTHS 
ANO  WEAKNESSES  OF  TRAINING.  DOCTRINE.  EQUIPMENT. 

ANO  ORGANIZATION  AS  THEY  RELATE  TO  MISSION  OUTCOMES. 


ALLOWS  FOR  THE  IDENTIFICATION  OF  PREDICTOR 
VARIABLES  WHICH.  IN  FACT.  ARE  RELATED  TO  MISSION 
EFFECTIVENESS. 


IDENTIFIES  TASKS  WHICH  ARE  HIGH-PAY-OFF  ACTIVITIES 
AS  THEY  RELATE  TO  MISSION  OUTCOME. 


ESTABLISHES  STANDARDS  FOR  CRITICAL  TASKS. 


ALLOWS  TRAINERS  TO  PRIORITIZE  TRAINING  REQUIRE¬ 
MENTS  NOW  THAT  THEY  ARE  AWARE  OF  THE  RELATIVE 
IMPORTANCE  OF  THOSE  REQUIREMENTS  TO  MISSION 
ACCOMPUSHMENT. 


FOCUSES  PERFORMANCE-OATA-GATHERING  EFFORTS  ON 
HIGH-PAY-OFF.  MISSION  CRITICAL  ACTIVITIES  WHICH 
PROVIDE  MEANINGFUL  "LESSONS  LEARNED." 


of  proposed  measurement  system 
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APPENDIX  A 
MILESTONE  CHART 


A-l 


MILESTONES 


REQUtREMENT  1.  DEVELOPMENT  OF  A  SYSTEM  FOR  MEASURING  AND 
EXPLAINING  UNIT  PERFORMANCE  EFFECTIVENESS  AT 
THE  NTC. 


Task  1:  Development  of  Combat  Mission  Effectiveness  Criterion  Variables. 
Subtasks 

1  1:  Davalop  Msssuramant  Modal  for  Evaluating  Mission 
Rastrits  at  ths  NTC 

1.2:  Establish  Mission  Parformanca  Conditions. 

1.3:  Sat  Mission  Performance  Standards. 

1.4:  Datarmina  Msssuramant  Requirement*  which  will  Raault 
in  Maasuras  of  Mission  Parformanca. 

1.S:  Design  Mission  Crltarion  Parformanca  Effactivanaaa  Indie  as. 
1.S:  Tast  Maasuramant  System  Faasibility. 

1.7:  Prapara  Technical  Report. 

Personnel  Resource  Requirements . . . . . . 

Task  2:  Development  of  Critical  Task  Parformanca  Maasuramant  Systam. 
Subtasks 

2.1:  Davalop  a  Msssuramant  Modal  for  Evaluating  Critical 
Task  Parformanca  at  the  NTC. 

2-2:  Analyze  and  Identify  Mission  Critical  Tasks  for  all  NTC 
Missions  from  Battalion  down  to  Platoon  Laval 
2.3:  Establish  Critical  Task  Conditions. 

2.4:  Establish  Critics!  Task  Parformanca  Standards. 

2.8  Datarmina  Msssuramant  Raquiramants  which  will  Result 
in  the  Measures  of  Critical  Task  Parformanca. 

2.8:  Develop  Critical  Task  Effectiveness  Criterion  Measures. 

2.7:  Propsre  Technical  Report. 

Personnel  Resource  Raquiramants . . 


REQUIREMENT  2.  DEVELOPMENT  OF  NTC-SPECIFIC  ARTEP  AND  OBSERVER 
GUIDES. 


Task  1:  Develop  NTC-Specific  ARTEP. 

Subtasks 

1.1:  Identify  Gspa  and  Potential  Inaccuracies  In  ths  Present 
ARTEP. 

1.2:  Produce  a  Draft  NTC- Specific  ARTEP. 

1.3:  Produce  Finalized  NTC-Spedfic  ARTEP 


Personnel  Resource  Requirements.. . 

Task  2:  Prepare  Observer  Guides. 

Subtssks 

2.1:  Develop  Format  for  Observer  Guides. 

2.2:  Produce  Draft  Observer  Guides. 

2.3:  Teat  Faasibility  of  Observer  Guides. 

2.4:  Prepare  Technical  Report  of  the  Field  Test  Rstuits. 

Personnel  Resource  Requirements . 


Figure  A-1.  Milestone  Chart 
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APPENOIX  B 


RESEARCH  PRODUCTS 


Listed  below  are  the  products  to  be  produced  in  the  accomplishment 
of  the  research  requirement  "Developing  and  Applying  Concepts  and  Methods 
for  Measurement  and  Interpretation  of  Unit  Performance  at  the  NTC."  The 
products  are  1 '  ;ted  in  chronological  order  under  their  respective 
research  requirements  and  tasks. 

REQUIREMENT  1:  DEVELOPMENT  OF  A  SYSTEM  FOR  MEASURING  AND  EXPLAINING  UNIT 
PERFORMANCE  EFFECTIVENESS  AT  THE  NTC. 

TASK  1:  DEVELOPMENT  OF  COMBAT  EFFECTIVENESS  CRITERIA. 

Product 


"A  MEASUREMENT  MODEL  FOR  EVALUATING  MISSION  RESULTS  AT  THE  NTC"  -  Final 
draft  due  in  May  86. 

o  Provides  a  structure  for  collecting  and  analyzing  mission  out¬ 
comes  in  terms  of  mission  conditions,  standards,  measures  of  performance, 
and  resultant  effectiveness. 


"PERFORMANCE  CONDITIONS  AT  THE  NTC"  -  Working  draft  due  in  Jul  86. 

o  An  analysis  (and  identification)  of  performance  conditions  for 
all  scenarios  that  are  exercised  at  the  NTC. 

"ESTABLISHMENT  OF  MISSION  PERFORMANCE  STANDARDS"  -  Working  draft  due  Oct 

86. 


o  Mission  performance  standards  will  be  established  by  presenting 
mission  situations  and  conditions  to  SMEs  and  aggregating  their  respon¬ 
ses,  and  applying  consensus  seeking  methodology  to  gain  group  acceptance 
of  the  standards.  Appropriate  military  doctrinal  references  will  also  be 
used. 


"CRITERION  PERFORMANCE  EFFECTIVENESS  INDICES"  -  Working  draft  due  in  Nov 

86. 


o  Based  on  the  unit  performance  measurement  model,  a  methodology 
will  be  developed  for  converting  performance  measures  into  mission  per¬ 
formance  effectiveness  indices. 


"A  STRAWMAN  PERFORMANCE  MEASUREMENT  SYSTEM"  -  Working  draft  due  in  Dec 

86. 


o  A  prototype  measurement  system  based  upon  previous  eight-month 
research  effort  will  be  presented  to  CATA  for  field  testing  and  evalua¬ 
tion. 


"A  MISSION  PERFORMANCE  MEASUREMENT  SYSTEM  FOR  THE  NTC"  -  Refined  system 
due  in  Feb  87. 

o  Final  version  of  the  measurement  system  will  be  based  on  CATA 
field  test  results  and  comments. 


"TECHNICAL  REPORT  ON  THE  MISSION  PERFORMANCE  MEASUREMENT  SYSTEM"  -Working 
draft  due  in  Apr  87. 

o  A  documented  report  on  the  mission  measurement  model  and  meth¬ 
odology  developed  for  its  use  at  the  NTC. 


TASK  2:  DEVELOPMENT  OF  CRITICAL  TASK  PERFORMANCE  MEASUREMENT 

SYSTEM.' 

Product 

"A  MEASUREMENT  MODEL  FOR  EVALUATING  CRITICAL  TASK  PERFORMANCE  AT  THE  NTC" 
-  Final  draft  due  in  May  86. 

o  Provides  a  structure  for  collecting  and  analyzing  performance 
of  critical  mission  tasks  in  terms  of  tasks,  conditions,  standards  and 
resultant  effectiveness. 


"MISSION  CRITICAL  TASKS  AT  THE  NTC"  -  Working  draft  due  Sep  86. 

o  An  analysis  (and  identification)  of  the  mission  critical  tasks 
for  each  of  the  NTC  missions  from  battalion  to  platoon  level. 


"ESTABLISHMENT  OF  CRITICAL  TASK  PERFORMANCE  CONDITIONS  AND  STANDARDS"  - 
Working  draft  due  in  Dec  86. 

o  Based  on  critical  tasks  established  earlier,  conditions  (situa¬ 
tions)  and  standards  will  be  established  by  SMEs  and  information  from 
appropriate  doctrinal  literature  for  each  critical  task. 


r 


REQUIREMENT  2:  DEVELOPMENT  OF  NTC-SPECIFIC  ARTEP  ANO  OBSERVER  GUIDES 
TASK  1:  DEVELOPMENT  OF  NTC-SPECIFIC  ARTEP. 

Product 

"GAPS  AND  POTENTIAL  INACCURACIES  IN  THE  ARTEP*  -  Working  draft  due  in 
Feb  87. 

o  These  findings  will  be  based  upon  the  earlier  established  cri¬ 
tical  tasks,  conditions,  and  standards  for  each  of  the  NTC  missions. 
Resolution  of  these  discrepancies  will  result  in  an  NTC  unique  ARTEP. 


•A  DRAFT  SUPPLEMENT  ARTEP  FOR  THE  NTC"  -  Working  draft  due  in  Apr  87. 

o  A  draft  NTC  Supplement  for  the  ARTEP  will  be  presented  to  CATA 
for  evaluation  and  field  testing.  The  results  of  the  CATA  evaluations 
will  be  used  to  produce  a  refined  ARTEP  Supplement  for  NTC-specific 
training. 


•FINALIZED  VERSION  OF  NTC-SPECIFIC  ARTEP"  -  Final  draft  due  in  Jul  87. 

o  Finalized  NTC-specific  ARTEP  will  be  produced  based  on  CATA 
field  testing  and  evaluation. 


TASK  2:  PRODUCE  OBSERVER  GUIOES 
Product  • 

•FORMAT  FOR  OBSERVER  GUIDES"  -  Working  draft  due  in  Sep  87. 

o  Based  on  the  NTC-specific  ARTEP,  observer  guides  will  be  pre¬ 
pared  which  will  focus  the  OC  observations  on  the  critical  ARTEP  type 
tasks.  This  particular  requirement  will  be  to  provide  for  CATA  review  a 
format  for  these  guides. 


■A  STRAWMAN  OBSERVER  GUIOE"  -  Working  draft  due  in  Dec  87. 

o  Based  on  CATA  review  of  format,  above,  a  strawman  OC  observer 
guide  will  be  produced  for  testing  on  selected  issues. 


■A  TECHNICAL  REPORT  ON  THE  FIELO  TEST  RESULTS  OF  STRAWMAN  OBSERVER  GUIDE" 
-  Working  draft  due  in  Mar  88. 

o  The  strawman  observer  guide  will  be  subjected  to  field  testing 
and  evaluation.  The  results  will  be  reported  in  a  technical  report. 
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